Supplementary material:Bayesian Optimization with Tree-structured Dependencies

In this supplementary material, we provide additional experimental results along with some details about the inference in our model and its nonlinear extension. 1. Experiments In this section, we provide complementary experimental results. 1.1. Optimization of synthetic tree-structured functions In addition to the results presented in the core of the paper, we include simulations with the non-linear extension of tree, denoted by tree-nonlinear, where the shared parameters zp = [rv]v∈Vp are modeled in a non-linear fashion (see details in Section 3 of the supplementary material). Moreover, we consider as well settings where the shared variables have a quadratic dependency in the leaf objectives, on both balanced and unbalanced binary trees; see Figure 1 and Figure 2. The trees on Figure 1 are referred to as small balanced, while those on Figure 2 are referred to as small unbalanced. We consider also higher-dimensional versions of those, with a depth of 4, resulting in respectively 8 and 9 leaves whose constant shifts are respectively {a × 0.1}a=1 and {a × 0.1}a=1 (they are referred to as large balanced and large unbalanced ). As defined in the core paper, all the non-shared continuous variables xj’s are defined in [−1, 1], while the shared ones are in [0, 1]. This implies that the best function value will always be 0.1. Amazon, Berlin, Germany. Amazon, Cambridge, United Kingdom. Correspondence to: Rodolphe Jenatton , Cedric Archambeau , Javier Gonzalez , Matthias Seeger . Proceedings of the 34 th International Conference on Machine Learning, Sydney, Australia, PMLR 70, 2017. Copyright 2017 by the author(s). We can see that the observations made in the core of the paper are still valid for the unbalanced trees. In particular, with linearly-dependent shared variables, tree-nonlinear converges more slowly than the linear version tree (middle panels of Figures 3 and 4), which may be the price to pay for having higher-dimensional latent variables c. Moreover, in presence of quadratically-dependent shared variables (right panels of Figures 3 and 4), we observe that tree fails to model adequately the non-linearities, while tree-nonlinear, as expected, can. In absence of shared variables (left panels of Figures 3 and 4), tree and tree-nonlinear are simply equivalent, which explains why their curves are superimposed. Finally, as dimension increases (i.e., going from the small to large settings), the performance of independent worsens, while that of tree and tree-nonlinear are barely affected. 1.2. Multi-layer perceptron tuning We report in Figure 5 the results of all methods including the non-linear extension of tree. The setting is identical to that described in the core of the paper. We can see that the linear version tree performs better than tree-nonlinear. This conclusion, perhaps surprising, is in good agreement with the recent observations from Zhang et al. (2016) where simple linear models outperformed more sophisticated, nonlinear competing methods within the context of the optimization of data analytic pipelines. 2. Details about the Tree-structured semi-parametric Gaussian process regression model We start by providing details about the posterior inference. 2.1. Posterior Inference The joint distribution P (y,g, c) of our model is given by P (c) ∏ pN (gp;bp,Kp)N (yp;gp + Z > p c, σ Inp), (1) where Kp = [Kp(xi,xj)]i,j∈Ip are kernel matrices, with the prior P (c) = N (c;0,Σc). Our goal is to obtain the posterior process P (gp(·)|c,yp) and the posterior distribution Supplementary material: Bayesian Optimization with Tree-structured Dependencies x1 x2 x24 + 0.1 x 2 5 + 0.2 x3 x26 + 0.3 x 2 7 + 0.4 0 1 0 1 0 1